WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCX 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification : 
G06K 9/00 



Al 



(11) International Publication Number: WO 00/10116 

(43) International Publication Date: 24 February 2000 (24.02.00) 



(21) International Application Number: PCmJS99/ 18542 

(22) International Filing Date: 13 August 1999 (13.08.99) 



(30) Priority Data: 
09/133,221 



13 August 1998 (13.08.98) 



US 



(71) Applicant: LAU TECHNOLCX5IES [USAJS]; 30 Porter Road. 
Littleton, MA 01460 (US). 

(72) Inventors: CUSACK, Francis, J„ Jr.; 32 Castle Drive. Groton. 

MA (US). EHN, Dennis. C; 215 Cypress Street, Newton 
Centre. MA (US). BORTOLUSSI. Jay; 21 Noel Road. 
Andover, MA (US). 

(74) Agents: LOREN, Ralph, A. et al.; Lahive & Cockficld. LLP, 
28 State Street, Boston, MA 02109 (US). 



(81) Designated States: AE, AL, AM. AT. AU. AZ. BA, BB. BG. 
BR, BY. CA, CH. CN. CR. CU, CZ, DE, DK. DM. EE, 
ES, FI. GB. GD. GE. GH, GM, HR. HU. ID, IL, IN, IS, JP, 
KE, KG. KP. KR. KZ, LC, LK, LR, LS, LT, LU» LV. MD. 
MG, MK, MN, MW, MX, NO, NZ, PL, PT, RO. RU, SD, 
SE, SG, SI, SK. SL, TJ, TM, TR, TT, UA, UG, UZ, VN, 
YU. ZA. ZW, ARIPO patent (GH, GM, KE, LS, MW, SD. 
SL. SZ, UG, ZW). Eurasian patent (AM. AZ. BY, KG. ICZ, 
MD, RU. TJ, TM), European patent (AT, BE, CH, CY, DE, 
DK, ES, Fl. FR. GB. GR. IE, IT, LU. MC, NL, PT, SE), 
GAP! patent (BF. BJ, CF, CG, CI. CM. GA. GN, GW, ML. 
MR, NE, SN. TD, TG). 



Published 

With international search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) Title: METHOD AND SYSTEM FOR ELIMINATING UNWANTED SHADOWS ON A SUBJECT IN A FACIAL RECOGNITION 
SYSTEM 



visroLE 

LIGHT 
SOURCE 



12 



16 




L 



FILTER 



18 



CAMERA 




COMPXJTER 
SYSTEM 



VACIAL 
RECOG. 
FACILITY 



20 



(57) Abstract 

A subject is illuminated by infrared light from an infrared light source. The subject may also be illuminated by visible light from 
ambient light sources. The infrared light helps to fully illuminate the subject so as to eliminate shadows on the face of the subject. A facial 
image of the subject is obtained and passed through a filter. The filter removes wavelengths of light in the visible range of the spectrum. 
The resulting image is digitized and passed to a facial recognition system. The filtered and digitized image is used to attempt to recognize 
the subject based on upon the facial image. 



•3 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madckgascar 


TJ 


Tajikistan 


BE 


Belgium 


CN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Vict Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


Zimbabwe 


CI 


C6te d'tvoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


' PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






cu 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federatbn 






DE 


Germany 


U 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






BE 


Estonia 


LR 


Liberia 


SC 


Sing^re 







wo 00/101 16 PCT/US99/18542 

- 1 - 

METHOD AND SYSTEM FOR ELIMINATING UNWANTED SHADOWS ON A 
SUBJECT IN A FACIAL RECOGNITION SYSTEM 



Technical Field 

The present invention relates generally to image acquisition systems and more 
particularly to a method and system for eliminating shadows on a subject in a facial 
recognition system. 



10 Background of the Invention 

Facial recognition systems attempt to identify a party based upon a facial image 
of the party. The facial image is typically obtained by a video camera that images the 
party and the facial image obtained by the video camera is compared with a database of 
15 facial images to determine if there are any matching facial images in the database. 
Identity information is stored along with the faces in the database. As such, once a 
match is found, the stored information may be accessed to retrieve the identity 
information £ind determine the identity of the subject. 

20 One difficulty with such conventional facial recognition systems relates to the 

acquisition of the facial image of the party. Lighting conditions may vary dramatically 
such that the lighting present when the facial image was obtained differs substantially 
from the lighting that was present when a corresponding image of the party stored in the 
database was obtained. One particular area of concern relates to shadows. Often the 

25 ambient light is poorly positioned and/or insufficient such that shadows appear on a 
party's face. These shadows may result in improper matching of the party with facial 
images in the database. One solution has been to brightly illuminate the party; hence, 
removing the shadows on the facial image of the party. Such bright illumination by 
visible light may cause the party to squint and/or result in discomfort to the party. 



30 
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Summary of the Invention 



The present invention addresses the limitations of the prior art by eliminating 
shadows on a face of a subject in a fashion that does not cause the subject to squint and 
5 does not resuh in discomfort to the subject. In one embodiment, the present invention 
employs one or more near-infrared light sources to illuminate a subject. It is presumed 
that a subject is illuminated in part by ambient visible light from another light source. A 
camera, such as a monochrome video camera, is employed to receive an image of the 
subject. The camera preferably is responsive to both the visible portion of the spectrum 

10 and the infrared portion of the spectrum. The filter is positioned to filter the image of 
the subject prior to receipt by the camera. The filter removes the visible spectral 
components in the image so that spectral components in the infrared wavelength range 
are passed to the camera. The camera then digitizes the image and passes the image 
onto a facial recognition component. The facial recognition component may be realized 

15 as a computer system that is programmed to implement a given facial recognition 
algorithm. 

The light source may be realized as an infrared light emitting diode or an infrared 
emitting lightbulb. Multiple light sources may be utilized to minimize the potential for 
20 shadows. Moreover, the light sources may be integrated into the camera, as may the 
filter. 



This embodiment of the present invention eliminates discomfort to the subject 
and eliminates squinting by the subject by illuminating the subject with light in a portion 
25 of the spectrum that is invisible to the subject. However, since the camera is responsive 
to the light emitted from the source, the camera is able to develop a complete image of 
the subject with light emitted from the light source. This embodiment has the additional 
benefits of being implemented both cheaply and efficiently. 
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An illustrative embodiment of the present invention will be described below 
relative to the following drawings. 

5 

FIGURE 1 depicts a system that is suitable for practicing the illustrative 
embodiment of the present invention in which the light source and filter are separate 
from the camera. 

10 FIGURE 2 depicts a system that is suitable for practicing the illustrative 

embodiment of the present invention wherein the light source is integrated into the 
camera. 

FIGURE 3 depicts a system that is suitable for practicing the illustrative 
1 5 embodiment of the present invention wherein the light source and the filter are 
integrated into the camera. 

FIGURE 4 depicts a system that is suitable for practicing the illustrative 
embodiment of the present invention wherein multiple light sources are utilized. 

20 

FIGURE 5 is a flow chart that illustrates the steps that are performed to eliminate 
shadows in the illustrative embodiment of the present invention. 

Description of the Illustrated Embodiment 

25 

The illustrative embodiment of the present invention illuminates a subject with 
one or more light sources that emit light in the infrared (particularly the near-infrared) 
portion of the radiation spectrum. These light sources help eliminate shadows that might 
otherwise be visible when only ambient visible light is present. Since the light sources 
30 emit light that is invisible to the human eye, the light emitted firom the light sources does 
not cause discomfort and does not result in a subject squinting. The video camera that is 
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used to obtain one or more images of the subject is preferably responsive to both visible 
light and infrared light. A filter is employed to remove spectral components in the 
visible range so that an infrared facial image is received by the camera and digitized. 
The quality of the resulting image is high and readily facilitates use of the acquired 
5 image in a facial recognition system. This approach of using the IR responsive camera 
in conjunction with an IR bandpass filter produces superior quality images relative to 
those obtained using a conventional camera configuration. 



In the illustrative embodiment of the present invention, the filtered and digitized 

10 facial image of a subject is passed to a facial recognition component. The facial 

recognition component may be employed, for example, in a computer system that is 
programmed to recognize the subject based on the filtered and digitized facial image. A 
suitable facial recognition system is U.S. Patent Application No. 09/1 19,485, entitled 
Real-Time Facial Recognition and Verification System, filed on July 20, 1998. This co- 

15 pending application is explicitly incorporated by reference herein. The elimination of 
the shadows by the light sources helps ensure that shadows do not decrease the 
efficiency with which the facial recognition system recognizes subjects. The facial 
recognition system may include a database of facial images. The filtered and digitized 
facial image can be compared with the database of facial images. The facial images 

20 stored in the database can be either complete digital representations of the person or can 
be templates of people created in a high dimensional image space, such as eigenfaces. 
Examples of such images and representative facial recognition systems for use with the 
present invention are described in U.S. Patent No. 5,164,992 of Turk et aL, U.S. Patent 
No. 5,432,864 of Lu et al, and U.S. Patent No. 5,386,103 of DeBan et al, the contents 

25 of which are hereby incorporated by reference. 

Figure 1 illustrates a system that is suitable for practicing the illustrative 
embodiment of the present invention. The system obtains a facial image of a subject 10 
and attempts to recognize the facial image based upon a database of facial image data. 
30 The system may be implemented in a number of different environments. For example, 
the system may be implemented for automatic teller machines (ATM) to determine 
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whether the subject is who the subject purports to be and is authorized to access the 
ATM. Likewise, the system may be implemented outside a controlled access area to 
limit the access area to authorized personnel. Still further, the system can be employed 
in a card production environment where the person's image is placed on an identification 
5 card, such as a drivers license. 



In the implementation shown in Figure 1, a near-infrared light source 14 
provides illumination of the subject 10. The near-infrared light source 14 is preferably 
positioned so as to illuminate fiilly the face of the subject 10 and to compensate for any 
10 shadows that otherwise persist on the face of the subject. It is further presumed that the 
subject 1 0 may be illuminated by a visible light source 12. The ambient visible light 
originates from a combination of sources. For example, the visible light sources might 
include the sun and light from light bulbs. 

15 Those skilled in the art will appreciate that the near infrared light source 14 can 

be implemented in a number of different ways. For example, the near-infrared light 
source 14 may be realized as a light emitting diode. Alternatively, the near-infrared 
light source 14 may be implemented using a halogen light bulb and a filter that blocks 
light having wavelengths shorter than 650 nanometers (or alternatively 700 nanometers). 

20 Those skilled in the art will appreciate that the near-infrared light source 14 may be 

positioned at different distances relative to subject 10. Preferably the near-infrared light 
source 14 is positioned a significant enough distance from the person so as to fully 
illuminate the face of the subject 10. The near-infrared light source 14, however, is not 
positioned so far away from the subject 10 so as to only weakly illuminate the subject. 

25 The near-infrared light source 14 is also positioned at an angle so that light reflected off 
the face of the subject 1 0 is received by the camera 1 8. Those skilled in the art will 
appreciate that a number of different angular orientations may be utilized. 



30 



The light reflecting off the face of the subject 10 passes through a filter 16. The 
filter 16 blocks selective spectral components of the reflected radiation from passing to 
or impinging on the camera 18. In particular, the filter blocks reflected spectral 
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coniponents within the visible range, i.e. wavelengths approximately between 400 to 700 
nanometers. Thus light in the infrared range, i.e. between about 0.65 micrometers and 
about 1 .00 micrometers, are permitted to pass to the camera 1 8. This includes 
wavelengths in the near-infrared range of between about 0.65 micrometers and about 2.0 
5 micrometers. 



The filter 16 can take many forms. For example, the filter can be a physical filter 
that includes materials for blocking light in the visible portion of the spectrum. 
Alternatively, the filter can be implemented programmatically to remove the visible 
10 components from the image of the subject that is received to the camera 1 8. Those 
skilled in the art will appreciate that many different types of filters may be utilized to 
realize the desired functionality. For example, a black glass filter of Schott RG 9 glass, 
catalog number 03 FCG 515 from Melles Griot may be used. 

15 The filter 16 may be positioned directly in front of the lens of the camera 18 to 

ensure that the image received by the camera 1 8 is properly filtered. Alternatively, the 
filter 16 may be positioned at other angles relative to the camera based upon the 
positioning of the light sources 12 and 14 and the subject 10. In Figure 1, the filter 16 is 
shown as being separate from the camera 1 8. 

20 

Those of ordinary skill will readily recognize that the camera 18 can take many 
forms. Preferably, the camera 1 8 is a video camera that records at a video frame rate 
(i.e. 30 frames per second). In the illustrative embodiment of the present invention it is 
presumed that the camera 1 8 is a monochrome camera that is infrared capable (i.e. it has 

25 an extended infrared (IR) response). An example of such a camera is the Cohu 4810 
with IR response. Those skilled in the art will appreciate that the present invention may 
also be practiced with color cameras and with cameras that do not have an extensive IR 
capability. One of the benefits of using the IR capable cameras is that they operate more 
effectively at lower levels of light, and particularly outdoors due to the sharply falling 

30 spectrum content of sky light. The fall off at longer wavelengths implies that the 
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illumination by the near-infrared light source 14 may be effective in the shade without 
having to resort to extremely bright light sources. 

With further reference to Figure 1 , the illustrated camera 1 8 can also be 
5 connected to a computer system 20. The computer system 20 can be a workstation, 
personal computer, mainframe computer, mini-computer or other computing system. 
The computer system 20 includes a facial recognition facility 22. The facial recognition 
facility 22 may be implemented in software to perform a given facial recognition 
algorithm, as discussed above. The computer system 20 receives a facial image in 
1 0 digital form from the camera 1 8. 

A number of different configurations are available for practicing the present 
invention. As shown in Figure 2, a light source 24 may be integrated into the camera 18. 
Moreover, the light source 24 need not emit light in the near-infrared range but may, 
15 instead, emit light in other portions of the infrared range. As shown in Figure 3, the 
filter 16 may also be directly integrated into the camera. Still further, as shown in 
Figure 4, multiple light sources 24A and 24B may be utilized. Preferably, the light 
sources 24 A and 24B are positioned so as to fully illuminate the face of the subject 10. 
Those skilled in the art will appreciate that more than two light sources may be utilized. 

20 

Figure 5 provides a flow chart that summarizes operation of the system for 
practicing the illustrative embodiment of the present invention. Initially, the subject 10 
. is illuminated with visible light and infrared light (step 30 in Figure 5). Those skilled in 
the art will appreciate that it is not necessary for the subject to be illuminated with the 

25 visible light to practice the present invention. The facial image of the subject produced 
by the light that reflects off the face of the subject from the infrared light source and the 
visible light passes through the filter 16 (step 32 in Figure 5). The filter removes light 
within the visible range of the spectrum. Thus, the light that passes through the filter 1 6 
onto the camera 1 8 is in the infrared portion of the spectrum. The resulting filtered 

30 facial image of the subject is received at the camera 1 8 (step 34 in Figure 5). As has 
been discussed above, the camera 1 8 digitizes the facial image and the digitized filtered 
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facial image is passed on to the computer system 20 where the facial recognition system 
22 performs facial recognition to attempt to identify the subject (step 36 in Figure 5). 

While the present invention has been described with reference to an illustrative 
5 embodiment thereof, those skilled in the art will appreciate that various changes as in 
form and detail may be made without departing from the intended scope of the present 
invention as defined by the appended claims. For example, three or more infrared light 
sources may be utilized and an array of infrared light emitting diodes may be used. 
Moreover, multiple visible light sources may be present to provide ambient light for 
1 0 illuminating the subject. 
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Claims 

1 . A system for obtaining and filtering an image of a subject exposed to variable 
lighting conditions, comprising: 

5 an infrared light source for illuminating a subject that is at least partially 

illuminated by visible light from another source with light in the infrared light 
wavelength range; 

a camera for receiving an image of the subject; and 
a filter for filtering the image of the subject having multiple spectral 
10 components before the image is received by the camera to remove spectral 

components in the image that lie substantially outside of the infrared light 
wavelength range. 

2. The system of claim 1 wherein the infrared light source is positioned on or about 
15 the camera. 1 

3. The system of claim 1 further comprising an additional infrared light source for 
further illuminating the subject. 

20 4. The system of claim 1 wherein a plurality of infrared light sources are positioned 
on the camera. 

5. The system of claim 1 wherein the camera comprises a monochrome camera 
with an extended infrared response. 

25 

6. The system of claim 1 wherein the filter passes spectral components in the 
infrared light wavelength range. 

7. The system of claim 1 wherein the camera comprises a video camera. 

30 

8. The system of claim 1 wherein the filter is integrated into the camera. 
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9. The system of claim 1 further comprising a facial recognition component for 
recognizing the subject based on the image of the subject received by the camera. 

5 10. The system of claim 1 wherein the infrared light source comprises a light 
emitting diode. 

1 1 . The system of claim 1 wherein the infrared light source comprises a halogen 
light bulb and a filter that blocks spectral components that are not infrared. 

12. A facial recognition system, comprising: 
at least one light source for emitting infrared light to illuminate a face of 

a subject; 

a filter for permitting non-visible spectral components in an image of the 
face of the subject to pass; 

a video camera that is responsive to visible light and to infrared light for 
receiving the spectral components in the image of the face of the subject that 
pass through the filter to produce a facial image of the subject; and 

a facial recognition component for receiving from the video camera and 
processing the facial image of the subject to recognize the subject. 

13. The facial recognition system of claim 12 wherein multiple light sources are 
used. 

25 14. The facial recognition system of claim 12 wherein the lights source is positioned 
on the video camera. 

15. The facial recognition system of claim 12 wherein the light sources emits 
infrared light. 
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16. The facial recognition system of claim 15 wherein the light source emits near- 
infrared light. 

17. The facial recognition system wherein the filter is integrated into the camera. 

5 

1 8. The facial recognition system of claim 12 wherein the facial recognition 
component comprises a computer system programmed to perform facial recognition. 

19. The facial recognition system of claim 12 wherein the video camera is a 
1 0 monochrome video camera with an infrared response. 

20. In a facial recognition system having a video camera response to visible light and 
infrared light a method of illuminating a subject to eliminate shadowing; comprising the 
steps of: 

15 providing a near-infrared light source; 

illuminating the subject with near-infrared light from the near- infrared 
light source, wherein said subject is already partially illuminated by visible light; 

filtering a facial image of the subject having multiple spectral 
components, produced by illumination of the subject by both the visible light and 
20 the near-infrared light source, to remove visible spectral components and 

produce a filtered facial image; and 

receiving the filtered facial image at the video camera. 

21 . The method of claim 1 wherein the method further comprises the step of 

25 applying a facial recognition technique to the filtered facial image received at the video 
camera to recognize the subject. 
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